Using kernel level simulation techniques to improve application program robustness

ABSTRACT

A method and system for simulating system conditions at a kernel-level is provided. In one aspect, process identifiers of processes for which simulation is to be performed are transmitted along with simulation pattern or rules from a user-space to a kernel space. Emulator in the kernel space intercepts system calls invoked by processes running in the user space. If the system calls originated from the one or more processes for which emulation was to be performed, return results according to the simulation pattern are generated and returned to the calling process.

TECHNICAL FIELD

This application relates to computer systems, and more particularly, tousing kernel-level techniques to simulate various abnormal conditionsduring development testing to improve application robustness.

BACKGROUND

During black-box software testing, a wide range of scenarios is notevaluated due to the difficulties of simulating the system conditionsthat provide those scenarios. For instance, creating abnormal orhigh-load system conditions such as the unreliable network, memorypressure, or lack of disk space conditions in a real working computersystem may negatively impact all other projects that happen to use thesame system. Creating such conditions on a real working computer systemmay also cause damages to the real system, entailing additional costs inthe system management and administration. Accordingly, it would bedesirable to be able to simulate abnormal system conditions so thatsoftware modules may be tested more thoroughly against a wide range ofabnormal system conditions, yet other software or projects running onthe same system may continue to be run under the normal systemconditions.

SUMMARY

A method of simulating testing conditions at a kernel-level is provided.The method in one aspect includes intercepting an operating system callfrom an application at a kernel-level. In the kernel-level, adetermination is made as to whether the operating system call wasinvoked from a process that was identified for failure emulation. If theoperating system call was invoked from a process that was identified forfailure emulation, user loaded rules are consulted and results to theoperating system call according to the user loaded rules are generatedand returned to the calling application. If the operating system callwas not invoked from a process that was identified for failureemulation, a native operating system service routine associated with theoperating system call is called and normal processing takes place.

The system for simulating testing conditions at a kernel-level in oneaspect includes a user-space module operable to transmit one or moreprocess identifiers and one or more rules associated with the processidentifiers for emulating failure conditions at a kernel-level and akernel-level module operable to intercept system call, and furtheroperable to determine whether the system call was invoked from one ormore processes identified by the one or more process identifiers and ifthe system call was invoked from the one or more processes identified bythe one or more process identifiers, the kernel-level module furtheroperable to generate a return result according to the one or more rules,and if the system call was not invoked from the one or more processesidentified by the one or more process identifiers, the kernel-levelmodule further operable to call native operating system service routineassociated with the system call.

In one embodiment, the intercepting of the system calls by the emulatormodule at the kernel level is transparent to the processes that invokethe system calls.

Further features as well as the structure and operation of variousembodiments are described in detail below with reference to theaccompanying drawings. In the drawings, like reference numbers indicateidentical or functionally similar elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the logic of processing in the failure emulatorkernel module in one embodiment.

FIG. 2 is a flow diagram illustrating method for activating failuresimulation in one embodiment.

FIG. 3 is a block diagram illustrating the kernel-level components anduser-space setup utility components in one embodiment.

DETAILED DESCRIPTION

In one embodiment, a kernel-level module simulates test environments ona selective basis, for example, on a per process basis. FIG. 1illustrates the logic of processing in the failure emulator kernelmodule in one embodiment. At 102, when a process is called, it isdetermined whether the process is subject to failure emulatorprocessing. This may be done, for example, by checking the identity ofthe called process, and whether the process's identity was previouslydownloaded from the user-space and identified as being the process forfailure emulation.

At 104, if a failure emulator is to be used, for example, the process isidentified as a process for failure emulation as determined at 102,syscall-dependent pre-syscall processing is performed at 106.Pre-syscall processing may include maintaining any statistics that thefailure emulator may choose to provide and any emulator-specific logicsuch as checking a counter for emulating intermittent failures, forexample, a failure in 50 percent of calls can be approximated bycondition (count % 2)==0. There are various design approaches which willdepend on the particular system call, for example, a short read may beemulated by truncating the size of a read request before the call to theoriginal syscall handler (that is, as part of pre-syscall processing) orby just returning part of the buffer of a full read. In some cases, itmay not be necessary to call the original syscall handler at all.

At 108, the original syscall handler is called and the results from thecall is saved. At 110, post-syscall processing is performed. Examples ofpost-syscall processing include generating syscall result as provided byfailure rules, generating error codes as provided by failure rules, andupdating system call statistics. Maintaining system call statistics mayhelp setting up and conducting tests.

At 104, if failure emulator is not being used, at 112 the originalsyscall handler is called and the call results are saved. At 114,syscall returns to its caller with appropriate results and/or errorcodes.

In one embodiment, the kernel module uses kernel-intercept technologywhere system calls are intercepted. The result of a particular systemcall executed by an application under testing depends on a set of rulesdownloaded to the kernel module using a user-level binary, for example,a user-space set up utility module.

The user-level binary provides control over what is simulated and how,for instance, intermittent short reads, occasional failure of memoryallocations. Intermittent and occasional failures can be emulated bymaintaining a set of counters for each system call and using a type ofpseudo random number generator. The user level binary is used tocommunicate selected failure types and patterns to the kernel modulethat simulates them.

In one embodiment, failure characterization are system call-based, forexample, “make 50 percent of read calls from a process with pid 1211fail pseudo-randomly with a short read error”. Each failure can bedescribed by the system call, percentage of times it should fail andexact type of failure possible for the system call in question. Processidentifiers of the target processes (for example, process owner, groupowner, pid) can be downloaded to the failure emulator kernel module by aseparate API call. The failure patterns (rules) may be selected by theuser based on his scope of interest and what is available or implementedin the failure emulator kernel module.

The simulation may be done without requiring any modifications of theapplications being tested. Thus, it is possible to test the exactapplication binary before the application is released to customers.

In one embodiment, a testing person may activate failure simulation fora particular process or group of processes by issuing a command thatprovides process identification to the kernel module together with achosen set of failures and their patterns. FIG. 2 is a flow diagramillustrating a method for activating failure simulation in oneembodiment. At 202, the group attribute of the process, for which thefailure condition is to be emulated, is set to one particular group.This way, one particular group may have the group ownership of theexecutable file that spawns the process. For example, for a file callednetdaemon:

-   -   groupadd failtest    -   chgrp failtest netdaemon        make the file netdaemon owned by failtest.

At 204, the identities of the processes that are to fail are downloadedto the failure emulator module. For example, the command line:fem_control-i-g failtest will download the identities to the failureemulator module called fem_control. At 206, failure test patterns aredownloaded as follows: fem_control-c 3-t 1, where -c 3 requests call # 3(read), -t 1 requests read failure of type 1. Failure type 1 may, forexample, be short reads. At 208 failure emulation is started, forexample, by the following command: fem_control -a 1. Here, the parameter“-a 1” sets active flag on, enabling the kernel-level emulation module.

At 210, the returned test failure patterns may be observed, for example,to check how the process responds to the failure. For example, oncesimulation is activated, a tester may observe program behaviorcorrelating observations with requested type of failure. At 212, thefailure emulation may be stopped, for example, by a command: fem_control-a 0.

In one embodiment, the kernel-level simulation system disclosed in thepresent application includes kernel-level components and user-spacesetup utility components. FIG. 3 is a block diagram illustrating thekernel-level components and user-space setup utility components in oneembodiment. The kernel-level components may be a library staticallylinked into the kernel during the kernel build. The kernel-levelcomponents may also be dynamically loaded as a module.

In the following discussion, both types are referred to as a failureemulator kernel module. The arrows in FIG. 3 illustrate control/datapaths when failure emulator is active. In one embodiment, all servicecalls from user programs 302 304 go through the system call dispatch 306to the failure emulator kernel module 310 which then calls the originalsystem call handler 312. In one embodiment, the failure emulator moduleis completely transparent. The test pattern rules 308 are consulted foreach process to be tested to see what kind of failure is requested.

The user-space setup utility components communicate setup data to thekernel-level components. User-space setup utility 314 in one embodimentis a program that is used to set up and control the kernel emulationmodule 310. It communicates with the kernel emulation module 310 via itsown system call that is installed during the kernel module startup in aspare system call table slot 316. In one embodiment, the failureemulator API 315 is based around that system call. The user-space setuputility 314 parses the command line, sets up the parameters for andmakes an API call 315. The API call 315 communicates with the kernelemulation module 310 using the system call 316.

In one aspect, operating service calls from the user-level 302, 304 areintercepted at the system call table level 306, by replacing theaddresses of original system call handlers with addresses of functionsin the failure emulator kernel module 310, then modifies their behaviorfor the calling processes, consulting the rules 308 uploaded by theuser-space control utility.

A typical system call wrapper in the failure emulator kernel module hascode similar to the following for a read system call: if (caller is atarget_process) { if (rules[__NR_read].enabled) { rc =(*orig_read_sycall) (arg1, arg2, arg3) ; switch (rules[[__NR_read].type){ case SHORT_READ: rc >>= 1 ; SET_ERRNO(0) ; break ; case INTRD_READ: rc= −1 ; SET_ERRNO(EINTR) ; break ; default: rc = −1 ; SET_ERRNO(EINTR) ;break ; } return rc ; } else { return (*orig_read_sycall) (arg1, arg2,arg3) ; } } else { return (*orig_read_sycall) (arg1, arg2, arg3) ; }

For all other processes running on the same system, native operatingsystem service routines are processed normally as shown in the aboveexample.

The system and method of the present disclosure may be implemented andrun on a general-purpose computer. The embodiments described above areillustrative examples and it should not be construed that the presentinvention is limited to these particular embodiments. Although thedescription was provided using the UNIX system call table as an example,it should be understood that the method and system disclosed in thepresent application may apply to other operating systems. Thus, variouschanges and modifications may be effected by one skilled in the artwithout departing from the spirit or scope of the invention as definedin the appended claims.

1. A method of simulating system conditions at a kernel-level,comprising: intercepting an operating system call from an application ata kernel-level; determining whether the operating system call was calledfrom a process that was identified for failure emulation; if theoperating system call was called from a process that was identified forfailure emulation, consulting user loaded rules and returning results tothe operating system call according to the user loaded rules; and if theoperating system call was not called from a process that was identifiedfor failure emulation, calling a native operating system service routineassociated with the operating system call.
 2. The method of claim 1,wherein the determining includes at least determining whether anidentifier of the process was communicated previously as being a processfor failure emulation.
 3. The method of claim 1, wherein the user loadedrules include type of failure to emulate, frequency of failure, or errorcodes to return, or combination thereof.
 4. The method of claim 1,wherein the intercepting is transparent to the process for failureemulation.
 5. A method of simulating system conditions at akernel-level, comprising: identifying one or more processes to akernel-level module for which to emulate failures; transmitting one ormore failure rules to the kernel-level module, the one or more failurerules associated with the one or more processes; activating thekernel-level module; and running the one or more processes.
 6. Themethod of claim 5, wherein the identifying includes at leasttransmitting one or more process identifiers to the kernel-level module,the one or more process identifiers associated with respective one ormore processes for which failure emulation is to be performed.
 7. Themethod of claim 5, further including deactivating the kernel-levelmodule.
 8. A program storage device readable by machine, tangiblyembodying a program of instructions executable by the machine to performmethod of simulating system conditions at a kernel-level, comprising:intercepting an operating system call from an application at akernel-level; determining whether the operating system call was calledfrom a process that was identified for failure emulation; if theoperating system call was called from a process that was identified forfailure emulation, consulting user loaded rules and returning results tothe operating system call according to the user loaded rules; and if theoperating system call was not called from a process that was identifiedfor failure emulation, calling a native operating system service routineassociated with the operating system call.
 9. The program storage deviceof claim 8, wherein the determining includes at least determiningwhether an identifier of the process was communicated previously asbeing a process for failure emulation.
 10. The program storage device ofclaim 8, wherein the user loaded rules include type of failure toemulate, frequency of failure, or error codes to return, or combinationthereof.
 11. The program storage device of claim 8, wherein theintercepting is transparent to the process for failure emulation.
 12. Asystem for simulating system conditions at a kernel-level, comprising: auser-space module operable to transmit one or more process identifiersand one or more rules associated with the process identifiers foremulating failure conditions at a kernel-level; and a kernel-levelmodule operable to intercept system call, and further operable todetermine whether the system call was invoked from one or more processesidentified by the one or more process identifiers and if the system callwas invoked from the one or more processes identified by the one or moreprocess identifiers, the kernel-level module further operable togenerate a return result according to the one or more rules, and if thesystem call was not invoked from the one or more processes identified bythe one or more process identifiers, the kernel-level module furtheroperable to call native operating system service routine associated withthe system call.
 13. The system of claim 12, wherein the user-spacemodule further includes an application programming interface thatcommunicates with the kernel-level module.